Method, apparatus, and system for communicating information of selected objects of interest displayed in a video-chat application

ABSTRACT

A method, apparatus and system for sharing object information which executes a video-chat session between an agent and a mobile client using an application hosted by a server. There is presented a view in common executed by a server application to both the agent and the mobile client for viewing objects captured by the mobile client during the video-chat session. There is identified by the agent a viewed object captured by the mobile client where the captured objected has been viewed by the agent in the common view during the video-chat session. The captured objected may be selected by the agent during the video-chat session and the agent may retrieve information remotely about the selected object to send to the mobile client.

TECHNICAL FIELD

Embodiments of the subject matter described herein relate generally toon-demand real-time communication over peer-to-peer connections. Moreparticularly, embodiments of the subject matter relate to a method,apparatus and system to identify, capture and exchange viewed objectinformation in video-chat apps between a mobile client and agent.

BACKGROUND

Currently, on-demand in-app buttons are used to integrate live agentsupport directly in mobile apps and webpages. These on demand in-appbuttons enable mobile clients to facilitate fast engagements byproviding a convenient means of requesting from a mobile device in whichthe mobile client is operating on-demand services by merely actuatingthe on-demand in-app button on the mobile client. From this buttonactuation a request is executed in an instant to initiate an on-demandvideo-chat session with an agent remotely located. Hence, upon actuatingthe in-app button, a video-chat session is initiated between the mobileclient and agent, and a one-to-one seamless communication connection iscreated in one step. The mobile client can receive immediate real-timecustomer support in the video-chat session from the agent.

In addition, the on-demand in-app buttons allows for enhancedpersonalized viewing experiences from the mobile client by providing360° views between the agent and the mobile client in such on-demandin-app video-chat sessions. Additionally, in the 360° views of thein-app video-chat sessions, there is a one-to-one correspondence in theviewing experience between both parties, that is, the mobile clientviews substantially the same display of imagery as what the agent views.Nevertheless, even though there is a common or shared view displayed toboth parties, limitations become apparent in the ability of the kind ofinformation that can be identified, accessed, collectively discussed andshared between the parties. The shared viewing experience is notcapitalized upon, rather communications exchanged back and forth aregenerally limited to voice or text messaging between the parties and notsharing of additional information such as object images viewed in themobile client and agent displays.

Accordingly, it is desirable to capitalize on the shared viewingexperiences in such video-chat sessions between the agent and mobileclient by configuring the on-demand in-app application to allowselecting and sharing of object data. In one instance, it is desired forthe on-demand video-chat in-app to have touch functionality forselecting objects by the agent to be viewed by the mobile client. Inother instances, it is desired for enabling by touch functions, acollecting of related object information on the part of the agent duringan in-app video-chat session. In such instances, it is desired for theagent to identify and highlight objects displayed and further retrieveremote information about such identified objects to share in the sharedview or share by other means with the mobile client. For example, it isdesired to enable the agent to direct camera operation of the mobileclient so the agent can choose objects of interest displayed andhighlight such objects for better viewing by both parties and addlabeling and other nomenclature to the object of interest for furtherdiscussions.

It is desirable to exchange information using a multi-tenant platformfor sharing image database information during the in-app video-chat. Inan exemplary example, it is desired to configure the in-app video-chatapplications to enable the agent to access information from a databaseassociated with the multi-tenant platform relating to object imagesidentified during the viewing experience between both parties. Hence, itis desirable in the in-app video-chat session to allow for more types ofinteraction than simply the customary voice and text messaging whencommunicating on-demand. Such additional information exchanges aredirected to object images which can be generated by using cameraoperations available on devices of the mobile client. For example, it isdesired on the mobile client in conjunction with the 360° viewingoperations of the mobile client that additional interactive object imageoriented tools be integrated for use during the in-app video-chatsessions. For example, these tools are directed to assist agents toaccess camera operations of the client device and can enable the agentto immediately identify viewed objects of interest during the video-chatsession for further review, sharing, and information collection.

In addition, it is desirable to initiate computer vision softwareapplications by a customer agent when making a physical touch orselection of the object in the shared view and to execute the computervision software applications, which may be hosted by the server ormobile client, for detecting and determining image features of theobject around a touch point. In addition, it is desirable to apply colorchanges or transforms for better contrasting the selected image objectfrom the surroundings in the shared view. Also, it is desirable to applymachine learning applications to information about the image object bythe agent to get more information about the selected object to send tothe mobile client.

It is desirable to use a multi-tenant cloud-based architectures toimprove collaboration, integration, and community-based cooperationbetween agents within tenant organizations without sacrificing datasecurity. Multi-tenancy refers to a system where a single hardware andsoftware platform simultaneously supports multiple user groups (alsoreferred to as “organizations” or “tenants”) from a common data storageelement (also referred to as a “multi-tenant database”).

The multi-tenant design provides a number of advantages overconventional server virtualization systems. First, the multi-tenantplatform operator can often make improvements to the platform based uponcollective information from the entire tenant community. Additionally,because all users in the multi-tenant environment execute applicationswithin a common processing space, it is relatively easy to grant or denyaccess to specific sets of data for any user within the multi-tenantplatform, thereby improving collaboration and integration betweenapplications and the data managed by the various applications. Themulti-tenant architecture therefore allows convenient and cost effectivesharing of similar application features between multiple sets of users.

Furthermore, other desirable features and characteristics will becomeapparent from the subsequent detailed description and the appendedclaims, taken in conjunction with the accompanying drawings and theforegoing technical field and background.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the subject matter may be derived byreferring to the detailed description and claims when considered inconjunction with the following figures, wherein like reference numbersrefer to similar elements throughout the figures.

FIG. 1 is an exemplary functional diagram illustrating the communicationprocess between the mobile client and agent in accordance with anembodiment;

FIG. 2 is an exemplary mobile device diagram illustrating using a shareddisplay in the communication process of the object sharing between theagent and the mobile client in accordance with an embodiment;

FIG. 3 is an exemplary agent diagram illustrating using the shareddisplay in the communication process of the object sharing between theagent and the mobile client in accordance with an embodiment;

FIG. 4 is an exemplary flowchart illustrating a layout of the operationof the communication process of the object sharing of the mobile clientand agent in accordance with an embodiment;

FIG. 5 is an exemplary system diagram illustrating a system ofcomponents of the communication process using the shared display of theobject sharing of the mobile client and agent in accordance with anembodiment;

FIG. 6 is an exemplary flowchart illustrating a layout of the operationof the communication process using the shared display of the objectsharing of the mobile client and agent in accordance with an embodiment;

FIG. 7 is a schematic block diagram of a multi-tenant computingenvironment for use in conjunction with the communication process of theobject sharing of the mobile client and agent in accordance with anembodiment.

DETAILED DESCRIPTION

With reference to FIG. 1, FIG. 1 is an exemplary functional diagramillustrating the communication of the mobile client and agent process inaccordance with an embodiment. A cloud based network system 100including the components of a mobile device 10 communicating via anetwork cloud 40 to a server 45 for supporting a video-chat app whichoperates on-demand with communicating via the network cloud 40 to themobile device 10 and the hosted by server 45. The network cloud 40 caninclude interconnected networks including both wired and wirelessnetworks for enabling communications of the mobile device 10 via amobile client 20 to the agent 50 hosted by server 45. For example,wireless networks may use a cellular-based communication infrastructurethat includes cellular protocols such as code division multiple access“CDMA”, time division multiple access “TDMA”, global system for mobilecommunication “GSM”, general packet radio service “GPRS”, wide band codedivision multiple access “WCDMA” and similar others. Additionally, wirednetworks include communication channels such as the IEEE 802.11 standardbetter known as “Wi-Fi”, the IEEE 802.16 standard better known as“WiMAX”, and the IEEE 802.15.1 better known as “Bluetooth”. The networkcloud 40 allows access to communication protocols and applicationprogramming interfaces that enable real-time communication overpeer-to-peer connections. As an example, this may include protocols fromopen source software packages for real-time video-chat communicationsover a cloud based network system 100 as described here. The webreal-time Communication “WebRTC”, which is an open source softwarepackage for real-time voice and video on the web, can depending on theversion be integrated in the Chrome, IOS, Explorer, Safari and otherbrowsers for peer-to-peer communications. Additionally, WebRTC canenable in-app video-chat communications through different browsersthrough a uniform standard set of APIs. Hence, the cloud based networksystem 100 allows for access for the communication of information withproviders of WebRTC during the on-demand peer-to-peer communications inin-app applications such as a video-chat app used in a mobile client 20.

The mobile device 10 includes the mobile client 20 which use a mobilesoftware development kit “SDK” platform. This SDK platform can provideone step activation of an on-demand services via the in-app application35 such as shown here of the mobile client 20 and a button type userinterface “UI” of the in-app application 35 for activating the on-demandservice. The mobile device 10 may include any mobile or connectedcomputing device including “wearable mobile devices” having an operatingsystem capable of running mobile apps individually or in conjunctionwith other mobile or connected devices. Examples of “wearable mobiledevices” include Google Glass® and Android® watches. Additionally,connected device may include devices such as cars, jet engines, homeappliances, tooth brushes, light sensors, air conditioning systems.Typically, the device will have display capabilities such as a displayscreens and also may have associated keyboard functionalities or even atouchscreen providing a virtual keyboard and buttons or icons on adisplay. Many such devices can connect to the internet and interconnectwith other devices via Wi-Fi, Bluetooth or other near fieldcommunication (NFC) protocols. Also, the use of cameras integrated intothe interconnected devices and GPS functions can be enabled.

The mobile client 20 may additionally include other in-app applicationsas well as SDK app platform tools and further can be configurable toenable downloading and updating of SDK app platform tools. In addition,the mobile client 20 uses an SDK platform which may be configurable fora multitude of mobile operating systems including Android, Apple iOS,Google Android, Research in Motion's BlackBerry OS, Nokia's Symbian,Hewlett-Packard's webOS (formerly Palm OS) and Microsoft's Windows PhoneOS etc . . . .

The in-app application 35 of the mobile client 20 provided on the SDKplatform can be found and downloaded by communicating with an on-lineapplication market platform for apps and in-apps which is configured forthe identifying, downloading and distribution of apps which areprebuilt. One such example is the Salesforce AppExchange® which is anonline application market platform for apps and in-apps where thedownloading, and installing of the pre-built apps and components such asan in-app application 35 for the mobile client 20 with video-chatfeatures can be downloaded.

In addition, these on-line application market platforms include“snap-in” agents for incorporation in the pre-built apps that are madeavailable. The in-app application 35 may be configured as a “snap-in”agent where the snap-in agent is considered by the name to be a completeSDK packages that allows for “easy to drop” enablement in the mobileclient 20 or in webpages. An example of a “snap-in” agent is the“Salesforce® SOS” which is a snap-in agent hosted by the SalesforceServiceCloud® and enables an UI configured as a button to be embeddedinto the mobile client 20. The UI button provides an instantaneousconnection directly to the agent 50 and provides on-demand live audio orvideo support and on-display guided assistance. In this example, thecomplete SDK snap-in agent provides the in-app application 35 withone-touch access to a customer support agent, as well as display sharingwith the mobile client 20. That is, in this instance there is a customersupport agent provided in video-chat communications of the in-appapplication 35 on the mobile device 10. This is provided by theSalesforce ServiceCloud® that enables customer support agents to havevideo-chat face to face sessions with customers using the mobile client20 in real time.

The server 45 acts as a host and includes the agent 50 that isconfigured for access by an application platform 65. The applicationplatform 65 can be configured as a platform as a service (“Paas”) thatprovides a host of features to develop, test, deploy, host andmaintain-applications in the same integrated development environment ofthe application platform. Additionally, the application platform 65 maybe part of a multi-tenant architecture where multiple concurrent usersutilize the same development applications installed on the applicationplatform 65. Also, by utilizing the multi-tenant architecture inconjunction with the application platform 65 integration with webservices and databases via common standards and communication tools canbe configured. As an example, Salesforce ServiceCloud® is an applicationplatform residing on the server 45 that hosts the agent 50 and may hostall the varying services needed to fulfil the application developmentprocess of the agent 50. The Salesforce ServiceCloud® as an example, mayprovide web based user interface creation tools to help to create,modify, test and deploy different UI scenarios of the agent 50.

The application platform 65 includes applications relating to the agent50. The agent 50 is an application that communications with the mobileclient 20, more specifically provides linking via the WebRTC to themobile client 20 for video-chat communications. The agent 50 may includeother applications in communication for accessing a multi-tenantdatabase 55 as an example, in a multi-tenant database system. Inaddition, the agent 50 may configurable to include UIs to displayvideo-chat communication. In an exemplary embodiment, the display 60which present the same view in the agent user interface of theapplication of the agent. The Salesforce ServiceCloud® platform is anapplication platform 65 that can host applications of an agent 50 forcommunication with an in-app application 35 of the mobile client 20. Thedisplay 60 may include sensing touch features that allows customeragents to touch object images viewed by the customer agent in thedisplay. The customer agent by touch sends a signal to the in-appapplication 35 of the mobile client 20. The object is located thatencompasses the touch point using computer vision software applicationsthat uses computer vision techniques among them image smoothing and edgedetection to detect the edges of the object image for the objectlocation.

The display can be configured to be a shared display. For example, inSalesforce® SOS™ in-app functions can control display sharing by usingan object linked to display sharing settings on a manager, in this casea change may be performed in the mobile client 20 associated with thevideo-chat session.

With continuing reference to FIG. 1, the display 60 of the agent 50includes image layering functions of an object 75 selected by the agent.For example, the agent 50 selects the object 75 and an overlay 15 of theobject 75 is overlaid using the computer vision applications and shownon the display 60 so the agent 50 receives a visual notification of theobject 75 selected. Additionally, because the display 60 is shared, theobject 75 overlay is also displayed on the display 25 of the mobileclient 20. Given that both displays may not be of the same size, theobject 75 is scaled accordingly in either display, that is the object 75may appear scaled larger in the display 60 of the agent 50 or vice versain the display 25 of the mobile client 20. Hence, the overlay 15supports functionality both on the mobile client 20 and the agent 50.

Additionally, the overlay 15 is integrated into functions of the snap-inagent, that is it is the overlay 15 is part of the operation of thesoftware development kit “SDK” allowing the overlay to be embedded in amultitude of mobile in-app user interfaces and agent user interfaceshence, enabling the displays of either agent 50 or the mobile client 20to easily be configured to include the snap-in objects as overlay. Theobject display similar to the SOS button may be a floating displayallowing the mobile client or agent to re-position the overlay on thedisplay. The overlay of the object 75 may be coordinated with thedisplay 60 of the object 75 enabling the agent 50 or mobile client 20 tomove the object 75 on either of the shared views on both displays.

Additionally, the application platform 65 has access to other databasesfor information retrieval and include a knowledge database 70 where bothagents and customers alike can search for the answers they need. Theknowledge database 70 may be part of the multi-tenant databasearchitecture allowing for communication with agent 50 and other mobileclients. The knowledge database 70 may include an object imagerepository configured to the allow the agent 50 to browse forinformation relating to the object image and send that information tothe mobile client 20. In addition, the application platform 65 canaccess a multi-tenant database 55 which is part of the multi-tenantarchitecture. The multi-tenant database 55 allows for enterprisecustomer access and the application platform 65 may be given access tothe multi-tenant database dependent upon differing factors such as asession ID associated with the video-chat session.

With reference to FIG. 2, FIG. 2 is an exemplary mobile device diagramillustrating the shared display of the object sharing process betweenthe agent and the mobile client in accordance with an embodiment. Themobile device 200 includes the mobile client 215 which hosts the in-appapplication 210 which may be a “snap-in” agent with an UI configure likea button for initiating or terminating the video-chat communicationsfrom the mobile client 215, a display 225 with the button UI, an object235 within the display and the background of the object 230. While thedisplay 225 is illustrated with the object 235 and background of theobject 230, the display 225 may also include a UI of a customer agent,other types of media i.e. any kind of information that can be viewed oris transmittable by video-chat applications. The mobile client 215 mayreside on a host such as a mobile device 200 which is different andtherefore can be considered agnostic and configurable to the mobiledevice 200 which performs the hosting. Additionally, the mobile client215 can be configured to reside in part or present in part on otherinterconnected devices. An example of this multi-device hosting would beinterconnections of smart phones coupled with wearable devices were thedisplay maybe found on an interconnected device or both the mobile andinterconnected device.

With continuing reference to FIG. 2, the object 235 is shown on themobile client 215 before and after selection by the customer agent 205and the overlay 265 applied. The object 235 illustrated in conjunctionwith the display 225 and in-app application 210 maintains a constantratio of the object 235 to the display 225. In other words, even, if thebackground 230 changes, the size of the object 235 which is selectedwith respect to the display 225 is unchanged. When changing the distanceof the object 235 selected by a camera (not shown) of the mobile device200, the object 235 again remains static in size so long as the object235 was selected prior to change of the camera distance to the object235. In other words, the computer vision applications used in the object235 selection are not re-initiated upon movement of the camera. In thesame vain, the actual size of the object 235 viewed on both the mobileclient 215 and the customer agent 205 shared in display 225 will notchange in ratio to the screen size unless there is a movement of thephone away from the object and a re-selection of the object 235 andsubsequent re-execution of associated computer vision applications. Thecomputer vision application of block image capture is executed whenidentifying the object 235 by the customer agent 205, the block imagecapture of the object 235 on the display 245 will generally be a 1 to 1representation of the object 235 viewed by the camera of the mobileclient 215.

In other words, upon image capture additional scaling will not beimmediately performed by the computer vision applications unlesssettings associated with the object 235 capture process are modified.Hence, by keeping the ratio a 1 to 1 ratio, the customer agent 205 willsend an image of the object 235 which is the same in pixel ratio in thedisplay 225 to the mobile device 200. For example, if mobile device 200is physically by movement distanced from the object 235 both the object235 and an identifier, which is the overlay 265, of the object will getsmaller and subsequent pixel ratios will decrease. Hence, the ratio ofthe overlay 265, object 235 in the display 225 in both the mobile device200 and application platform (not shown) of the customer agent 205 actin coordination to maintain consistency in the object appearance to themobile client 215 and to the customer agent 205.

With reference to FIG. 3, FIG. 3 is an exemplary agent diagramillustrating using the shared display in the communication process ofthe object sharing between the agent and the mobile client in accordancewith an embodiment. FIG. 3 illustrates a server 300 that hosts anapplication platform 305 where a customer agent accesses applicationssuch as the agent 310 using an agent console 360 which includes a userinterface of the agent 310 configured to present a display 315 of thecommunications from the in-app application for video-chat of the mobileclient 370 and to access the knowledge database 330. An example of theapplication platform 305 that allows access to a knowledge database 330is the Salesforce ServiceCloud® platform which allows a customer agent,by authenticating video-chat session ID from the agent 310, to accessthe knowledge database 330. When an image search is performed of theobject 320, block image data captured from the object 320 as well asadditional data added by the customer agent may be used to search theknowledge database 330. In addition, meta data can be extracted from thedata packets associated with the captured block image data forsearching.

As an example of this search, the image search is performed using astructured query language “SQL” of the knowledge database 330 that maymatch header information tagged to the packet data of the captured imageblocks to identify related block images. When performing this search,call procedures that are used in the search algorithm are performed inbranches of trees of the knowledge database 330, the call procedures arenot randomly performed but follow paths that are enterprise specific toa customer. This manner of search would allow enterprise customers toprioritize images in the knowledge databases which are known to besearched often by pre-loading the images in a tree structure thatanticipates the image related information requests by an agent 310 to amobile client 370. In other words, information requested more often bythe agent 310 would be given higher weighting in structured database. Asan example, a block image of an object 320 associated with new productsmay be of greater interest and hence customer agent requests for relatedinformation could be discerned and the preloaded images added with suchpriority in mind.

Once the object 320 is selected a set of processes of computer visionapplications for image capture is initiated for capturing the images inblocks. This block capture of the image 345 can also be used in theoverlay process. A multi-stage processing is performed by calling aseries of procedures of computer vision applications to perform theimage capture of the selected image of the object 320 and extract theassociated packet data and create the block image. There are a host ofavailable libraries that provide such processing tools for such computervision applications.

Open source computer vision “OpenCV” is an example of one such libraryin which an open-source computer vision and machine learning softwareprocedures are available and may be called in the present image captureprocessing. For example, in OpenCV a series of routines related toconversion of the color transforms to grayscale, Gaussian blur, cannyedge detection, structuring of data elements, image dilation, andascertaining the object contours are available for use in the capturingprocesses.

BoofCV is another open source library for real-time computer visionapplications BoofCV is similarly organized into multiple types ofroutines for image processing, features, geometric vision, calibration,recognition, and input/output 10″. BoofCV like OpenCV for imageprocessing contains commonly used image processing functions whichoperate directly on pixels.

These computer vision applications also contain features such as thefollowing: features for extraction algorithms for use in higher leveloperations; features for calibration which are routines for determiningthe camera's intrinsic and extrinsic parameters; features forrecognition which are for recognition and tracking complex visualobjects; features for geometric vision which is composed of routines forprocessing extracted image features using 2D and 3D geometry; featuresfor visualize which has routines for rendering and displaying extractedfeatures; and features for 10 which is for input and output routines fordifferent data structures. A select subset of such features can be usedin the image processing steps of the present disclosure to create amongthings the block images, the overlay etc.

The information of the block of the image 345 is sent to the knowledgedatabase 330 for search by the customer agent. Information is gatheredfrom the search and may be overlaid on the block of the image 345 in theshared viewed 315 or may be sent in alternate channels such asSalesforce Chatter® to the mobile client 370. Additionally, the customeragent can update the knowledge database 330 with the information gatherduring the video-chat session.

Once the image is found, information associated with the image and aboutthe image will be passed back to the mobile client 370 for display onthe displays of the mobile client and the agent. The SOS SDK applicationhas a reset function that allows this kind of information to becustomizable by the agent each time it is sent. Hence, the addedinformation can be changed using the reset function to allow theinformation displayed on the image to be changed.

Referring to FIG. 4, FIG. 4 is an exemplary flowchart illustrating alayout of the operation of the communication process of the objectsharing of the mobile client and agent in accordance with an embodiment.Initially, a video-chat session is started 405 with a camera on the sideof the mobile client in operation. The camera may be part of the mobiledevice hosting the mobile client or may be part of an interconnecteddevice. Nevertheless, a camera is operated that is capable ofcommunicating and providing images to the display of the mobile clientand the display of the agent. Generally, the camera provides video inthe format of MPEG video streaming data but other similar alternativesmay also be used. In an exemplary embodiment, a Salesforce® SOS™ sessionwith a camera running is commenced at 405. The session connects with theagent 415 or in the alternative in the session, capture routines may beinitiated to capture all objects in the operated camera view 410 henceomitting the need for agent identification of the object 450.

Alternately, as mentioned, if the objects are not captured 410, then acustomer agent physically touches the display to identify and select anobject 450 in the camera view. Then in the following step 445, theobject is located that encompasses the touch point which is the physicalpoint identified by the agent. Procedures as described above from OpenCVor other open sources foundations are applied for object detection andblock image capture. Next at 420, a determination is performed if theobject is matched, that is if the touch location coordinates are matchedto an object. If not matched, then all the objects are captured at 410in the view to resolve the lack of match impediment. At 425, the imagedata is stored on the mobile device in non-volatile memory. Thenon-volatile memory may be random access memory “RAM” or programmableFLASH memory. In either case whether the object is matched or not at420, the flow proceeds to 430 were key information about the locatedobject is saved. Next, the flow proceeds in two separate process paths.In one path at 440 artificial intelligence techniques are appliedincluding machine learning to retrieve information about the object fromthe ServiceCloud at 455 and determine whether there is a match with animage already stored at the server at 460. If there is no match theinformation including session data is stored in the ServiceCloud 455.Alternatively, if there is a match, at 470 additional information isretrieved and sent to the mobile client. The additional information maybe sent and overlaid or displayed in a manner were it is associated withthe image at the display of the mobile client at 465, or in otherinstances, the information can be delivered to the mobile client in ahost of other communication paths such as Salesforce Chatter®, instantmessaging, email, or by various social networks.

With reference to FIG. 5, FIG. 5 is an exemplary system diagramillustrating a system of components of the communication process usingthe shared display of the object sharing of the mobile client and agentin accordance with an embodiment. A communication system 500 of themobile client and agent with the WebRTC service provider is illustrated.The communication system 500 includes a camera view 510 of the mobileclient 520 and the corresponding view of the agent 550 for video-chat.The camera view 510 of the mobile client at the mobile device is viewused for capturing objects at the agent 530 which are identified fromthe agent view which is as explained the camera view 510. Additionally,an application platform 590 supports hosts the agent 530 which includesthe UI for the agent 550 of the display and is connected to theknowledge database 560 which containing information about the images andobjects. Additionally, a multi-tenant image database 570 is connected tothe application platform 590 allowing for access based on a session IDidentifier of an enterprise client associated with the mobile client.The mobile client 520 includes the in-app video-chat application 535hosted by the mobile client 520 and includes image capture routines 540process by a processor 545 locally located at the mobile client. Thelocal processor can process all the binary data captured by the cameraconnected to the mobile client. The in-app application in one instancemay be Salesforce Service SOS® hosted by the Salesforce® SDK which canbe considered the in-app platform for the mobile client 520. The camerahaving display which is the camera view 510 is connected to the in-appplatform of the mobile client 520. The WebRTC provides real-timemultimedia applications (i.e. video-chat communication) on the web,without requiring plugins, downloads or installs. WebRTC consists ofseveral interrelated APIs and protocols which are arranged interminglingto enable signaling and connecting to a server from a different platformmobile device. The communication of information flow is sent 565bi-directionally to and from the WebRTC provider 525 to the mobileclient 520 and the agent 530.

The communication process between the mobile client 520 and the agent530 can be summarized in a multi-step process. In the initial step at570, communication to the application platform 590, the SalesforceServiceCloud® from the mobile client 520 is commenced. At the secondstep at 575, agent 530 selects an object from the mobile client 520. Inthe third step at 580, an image of the captured object is transferred tothe application platform. At the final step at 585, additionalinformation about the object is transferred from the applicationplatform 590 to the mobile client 520.

With reference to FIG. 6, FIG. 6 is an exemplary flowchart illustratinga layout of the operation of the communication process using the shareddisplay of the object sharing of the mobile client and agent inaccordance with an embodiment. the operation of the software proceduresof the mobile client and the agent are further described and illustratedin the agent/mobile client system 600. Initially, the view of the cameraconnected to the mobile device is initiated at 605. The view of thecamera is shared with the agent at 610. A customer agent using the agentphysically touches an object on the camera view at 620. In parallel, aset of computer vision applications is processed for enabling imagecapture locally by a processor of the mobile agent at 615. The computervision applications enable the locating of an object that encompassesthe physical touch of the customer agent at 620. Executing the computervision applications at 615, the processor of the mobile device at 625locates the object that encompasses a touch point which is theapproximate center of the customer agent physical touch.

Again, executing the computer vision applications at 615, the mobileclient at 630 captures all the surrounding images and also the greyscalevalues of the object which has been selected from the prior steps 620and 625. Here in addition to saving the surrounding data at an imagedatabase locally at the mobile client at 635 an overlay which is lightor translucent is applied at 640 covering the object to change the imagecolor of the object for easier identification from the background. Then,by using image data from the image data at the server which isconnectable by the application platform ServiceCloud® at 660, machinelearning and artificial intelligence techniques are applied at 655 toidentify the object from a collection of images from the image database660. Next, the existing information about the object is gathered at 650and displayed in the camera view at 645 so that the mobile client canview the existing information.

With reference to FIG. 7, FIG. 7 is a schematic block diagram of amulti-tenant computing environment for use in conjunction with thecommunication process of the object sharing of the mobile client andagent in accordance with an embodiment. A server may be shared betweenmultiple tenants, organizations, or enterprises, referred to herein as amulti-tenant database. In the exemplary disclosure, video-chat data andservices are provided via a network 745 to any number of tenant devices740, such as desk tops, laptops, tablets, smartphones, Google Glass™,and any other computing device implemented in an automobile, aircraft,television, or other business or consumer electronic device or system,including web tenants.

Each application 728 is suitably generated at run-time (or on-demand)using a common type of application platform 710 that securely providesaccess to the data 732 in the multi-tenant database 730 for each of thevarious tenant organizations subscribing to the service cloud 700. Inaccordance with one non-limiting example, the service cloud 700 isimplemented in the form of an on-demand multi-tenant customerrelationship management (CRM) system that can support any number ofauthenticated users for a plurality of tenants.

As used herein, a “tenant” or an “organization” should be understood asreferring to a group of one or more users (typically employees) thatshares access to common subset of the data within the multi-tenantdatabase 730. In this regard, each tenant includes one or more usersand/or groups associated with, authorized by, or otherwise belonging tothat respective tenant. Stated another way, each respective user withinthe multi-tenant system of the service cloud 700 is associated with,assigned to, or otherwise belongs to a particular one of the pluralityof enterprises supported by the system of the service cloud 700.

Each enterprise tenant may represent a company, corporate department,business or legal organization, and/or any other entities that maintaindata for particular sets of users (such as their respective employees orcustomers) within the multi-tenant system of the service cloud 700.Although multiple tenants may share access to the server 702 and themulti-tenant database 730, the particular data and services providedfrom the server 702 to each tenant can be securely isolated from thoseprovided to other tenants. The multi-tenant architecture thereforeallows different sets of users to share functionality and hardwareresources without necessarily sharing any of the data 732 belonging toor otherwise associated with other organizations.

The multi-tenant database 730 may be a repository or other data storagesystem capable of storing and managing the data 732 associated with anynumber of tenant organizations. The multi-tenant database 730 may beimplemented using conventional database server hardware. In variousembodiments, the multi-tenant database 730 shares the processinghardware 704 with the server 702. In other embodiments, the multi-tenantdatabase 730 is implemented using separate physical and/or virtualdatabase server hardware that communicates with the server 702 toperform the various functions described herein.

In an exemplary embodiment, the multi-tenant database 730 includes adatabase management system or other equivalent software capable ofdetermining an optimal query plan for retrieving and providing aparticular subset of the data 732 to an instance of application (orvirtual application) 728 in response to a query initiated or otherwiseprovided by an application 728, as described in greater detail below.The multi-tenant database 730 may alternatively be referred to herein asan on-demand database, in that the multi-tenant database 730 provides(or is available to provide) data at run-time to on-demand virtualapplications 728 generated by the application platform 710, as describedin greater detail below.

In practice, the data 732 may be organized and formatted in any mannerto support the application platform 710. In various embodiments, thedata 732 is suitably organized into a relatively small number of largedata tables to maintain a semi-amorphous “heap”-type format. The data732 can then be organized as needed for a particular virtual application728. In various embodiments, conventional data relationships areestablished using any number of pivot tables 734 that establishindexing, uniqueness, relationships between entities, and/or otheraspects of conventional database organization as desired. Further datamanipulation and report formatting is generally performed at run-timeusing a variety of metadata constructs. Metadata within a universal datadirectory (UDD) 736, for example, can be used to describe any number offorms, reports, workflows, user access privileges, business logic andother constructs that are common to multiple tenants.

Tenant-specific formatting, functions and other constructs may bemaintained as tenant-specific metadata 738 for each tenant, as desired.Rather than forcing the data 732 into an inflexible global structurethat is common to all tenants and applications, the multi-tenantdatabase 730 is organized to be relatively amorphous, with the pivottables 734 and the metadata 738 providing additional structure on anas-needed basis. To that end, the application platform 710 suitably usesthe pivot tables 734 and/or the metadata 738 to generate “virtual”components of the virtual applications 728 to logically obtain, process,and present the relatively amorphous data from the multi-tenant database730.

The server 702 may be implemented using one or more actual and/orvirtual computing systems that collectively provide the dynamic type ofapplication platform 710 for generating the virtual applications 728.For example, the server 702 may be implemented using a cluster of actualand/or virtual servers operating in conjunction with each other,typically in association with conventional network communications,cluster management, load balancing and other features as appropriate.The server 702 operates with any sort of processing hardware 704 whichis conventional, such as a processor 705, memory 706, input/outputfeatures 707 and the like. The input/output features 707 generallyrepresent the interface(s) to networks (e.g., to the network 745, or anyother local area, wide area or other network), mass storage, displaydevices, data entry devices and/or the like.

The processor 705 may be implemented using any suitable processingsystem, such as one or more processors, controllers, microprocessors,microcontrollers, processing cores and/or other computing resourcesspread across any number of distributed or integrated systems, includingany number of “cloud-based” or other virtual systems. The memory 706represents any non-transitory short or long term storage or othercomputer-readable media capable of storing programming instructions forexecution on the processor 705, including any sort of random accessmemory (RAM), read only memory (ROM), flash memory, magnetic or opticalmass storage, and/or the like. The computer-executable programminginstructions, when read and executed by the server 702 and/or processor705, cause the server 702 and/or processor 705 to create, generate, orotherwise facilitate the application platform 710 and/or virtualapplications 728 and perform one or more additional tasks, operations,functions, and/or processes described herein. It should be noted thatthe memory 706 represents one suitable implementation of suchcomputer-readable media, and alternatively or additionally, the server702 could receive and cooperate with external computer-readable mediathat is realized as a portable or mobile component or platform, e.g., aportable hard drive, a USB flash drive, an optical disc, or the like.

The application platform 710 is any sort of software application orother data processing engine that generates the virtual applications 728that provide data and/or services to the tenant devices 740. In atypical embodiment, the application platform 710 gains access toprocessing resources, communications interface and other features of theprocessing hardware 704 using any sort of conventional or proprietaryoperating system 708. The virtual applications 728 are typicallygenerated at run-time in response to input received from the tenantdevices 740. For the illustrated embodiment, the application platform710 includes a bulk data processing engine 712, a query generator 714, asearch engine 716 that provides text indexing and other searchfunctionality, and a runtime application generator 720. Each of thesefeatures may be implemented as a separate process or other module, andmany equivalent embodiments could include different and/or additionalfeatures, components or other modules as desired.

The runtime application generator 720 dynamically builds and executesthe virtual applications 728 in response to specific requests receivedfrom the tenant devices 740. The virtual applications 728 are typicallyconstructed in accordance with the tenant-specific metadata 738, whichdescribes the particular tables, reports, interfaces and/or otherfeatures of the particular application 728. In various embodiments, eachvirtual application 728 generates dynamic web content that can be servedto a browser or other tenant program 742 associated with its tenantdevice 740, as appropriate.

The runtime application generator 720 suitably interacts with the querygenerator 714 to efficiently obtain data 732 from the multi-tenantdatabase 730 as needed in response to input queries initiated orotherwise provided by users of the tenant devices 140. In a typicalembodiment, the query generator 714 considers the identity of the userrequesting a particular function (along with the user's associatedtenant), and then builds and executes queries to the multi-tenantdatabase 730 using system-wide metadata 736, tenant specific metadata,pivot tables 734, and/or any other available resources. The querygenerator 714 in this example therefore maintains security of the commondatabase by ensuring that queries are consistent with access privilegesgranted to the user and/or tenant that initiated the request.

With continued reference to FIG. 7, the bulk data processing engine 712performs bulk processing operations on the data 732 such as uploads ordownloads, updates, online transaction processing, and/or the like. Inmany embodiments, less urgent bulk processing of the data 732 can bescheduled to occur as processing resources become available, therebygiving priority to more urgent data processing by the query generator714, the search engine 716, the virtual applications 728, etc.

In exemplary embodiments, the application platform 710 is utilized tocreate and/or generate data-driven virtual applications 728 for thetenants that they support. Such virtual applications 728 may make use ofinterface features such as custom (or tenant-specific) screens 724,standard (or universal) screens 722 or the like. Any number of customand/or standard objects 726 may also be available for integration intotenant-developed virtual applications 728. As used herein, “custom”should be understood as meaning that a respective object or applicationis tenant-specific (e.g., only available to users associated with aparticular tenant in the multi-tenant system) or user-specific (e.g.,only available to a particular subset of users within the multi-tenantsystem), whereas “standard” or “universal” applications or objects areavailable across multiple tenants in the multi-tenant system.

The data 732 associated with each virtual application 728 is provided tothe multi-tenant database 730, as appropriate, and stored until it isrequested or is otherwise needed, along with the metadata 738 thatdescribes the particular features (e.g., reports, tables, functions,objects, fields, formulas, code, etc.) of that particular virtualapplication 728. For example, a virtual application 728 may include anumber of objects 726 accessible to a tenant, wherein for each object726 accessible to the tenant, information pertaining to its object typealong with values for various fields associated with that respectiveobject type are maintained as metadata 738 in the multi-tenant database730. In this regard, the object type defines the structure (e.g., theformatting, functions and other constructs) of each respective object726 and the various fields associated therewith.

Still referring to FIG. 7, the data and services provided by the server702 can be retrieved using any sort of personal computer, mobiletelephone, tablet or other network-enabled tenant device 740 on thenetwork 745. In an exemplary embodiment, the tenant device 740 includesa display device, such as a monitor, screen, or another conventionalelectronic display capable of graphically presenting data and/orinformation retrieved from the multi-tenant database 730, as describedin greater detail below.

Typically, the user operates a conventional browser application or othertenant program 742 executed by the tenant device 740 to contact theserver 702 via the network 745 using a networking protocol, such as thehypertext transport protocol (HTTP) or the like. The user typicallyauthenticates his or her identity to the server 702 to obtain a sessionidentifier (“Session ID”) that identifies the user in subsequentcommunications with the server 702. When the identified user requestsaccess to a virtual application 728, the runtime application generator720 suitably creates the application at run time based upon the metadata738, as appropriate. However, if a user chooses to manually upload anupdated file (through either the web based user interface or through anAPI), it will also be shared automatically with all of the users/devicesthat are designated for sharing.

As noted above, the virtual application 728 may contain Java, ActiveX,or other content that can be presented using conventional tenantsoftware running on the tenant device 740; other embodiments may simplyprovide dynamic web or other content that can be presented and viewed bythe user, as desired. As described in greater detail below, the querygenerator 714 suitably obtains the requested subsets of data 732 fromthe multi-tenant database 730 as needed to populate the tables, reportsor other features of the particular virtual application 728. In variousembodiments, application 728 embodies the functionality of aninteractive performance review template linked to a database ofperformance metrics, as described below in connection with FIGS. 1-6.

Techniques and technologies may be described herein in terms offunctional and/or logical block components, and with reference tosymbolic representations of operations, processing tasks, and functionsthat may be performed by various computing components or devices. Suchoperations, tasks, and functions are sometimes referred to as beingcomputer-executed, computerized, software-implemented, orcomputer-implemented. In practice, one or more processor devices cancarry out the described operations, tasks, and functions by manipulatingelectrical signals representing data bits at memory locations in thesystem memory, as well as other processing of signals. The memorylocations where data bits are maintained are physical locations thathave particular electrical, magnetic, optical, or organic propertiescorresponding to the data bits. It should be appreciated that thevarious block components shown in the figures may be realized by anynumber of hardware, software, and/or firmware components configured toperform the specified functions. For example, an embodiment of a systemor a component may employ various integrated circuit components, e.g.,memory elements, digital signal processing elements, logic elements,look-up tables, or the like, which may carry out a variety of functionsunder the control of one or more microprocessors or other controldevices.

When implemented in software or firmware, various elements of thesystems described herein are essentially the code segments orinstructions that perform the various tasks. The program or codesegments can be stored in a processor-readable medium or transmitted bya computer data signal embodied in a carrier wave over a transmissionmedium or communication path. The “processor-readable medium” or“machine-readable medium” may include any medium that can store ortransfer information. Examples of the processor-readable medium includean electronic circuit, a semiconductor memory device, a ROM, a flashmemory, an erasable ROM (EROM), a floppy diskette, a CD-ROM, an opticaldisk, a hard disk, a fiber optic medium, a radio frequency (RF) link, orthe like. The computer data signal may include any signal that canpropagate over a transmission medium such as electronic networkchannels, optical fibers, air, electromagnetic paths, or RF links. Thecode segments may be downloaded via computer networks such as theInternet, an intranet, a LAN, or the like.

“Node/Port”—As used herein, a “node” means any internal or externalreference point, connection point, junction, signal line, conductiveelement, or the like, at which a given signal, logic level, voltage,data pattern, current, or quantity is present. Furthermore, two or morenodes may be realized by one physical element (and two or more signalscan be multiplexed, modulated, or otherwise distinguished even thoughreceived or output at a common node). As used herein, a “port” means anode that is externally accessible via, for example, a physicalconnector, an input or output pin, a test probe, a bonding pad, or thelike.

The following description refers to elements or nodes or features being“connected” or “coupled” together. As used herein, unless expresslystated otherwise, “coupled” means that one element/node/feature isdirectly or indirectly joined to (or directly or indirectly communicateswith) another element/node/feature, and not necessarily mechanically.Likewise, unless expressly stated otherwise, “connected” means that oneelement/node/feature is directly joined to (or directly communicateswith) another element/node/feature, and not necessarily mechanically.Thus, although the schematic shown in FIG. 7 depicts one exemplaryarrangement of elements, additional intervening elements, devices,features, or components may be present in an embodiment of the depictedsubject matter.

In addition, certain terminology may also be used in the followingdescription for the purpose of reference only, and thus are not intendedto be limiting. For example, terms such as “upper”, “lower”, “above”,and “below” refer to directions in the drawings to which reference ismade. Terms such as “front”, “back”, “rear”, “side”, “outboard”, and“inboard” describe the orientation and/or location of portions of thecomponent within a consistent but arbitrary frame of reference which ismade clear by reference to the text and the associated drawingsdescribing the component under discussion. Such terminology may includethe words specifically mentioned above, derivatives thereof, and wordsof similar import. Similarly, the terms “first”, “second”, and othersuch numerical terms referring to structures do not imply a sequence ororder unless clearly indicated by the context.

For the sake of brevity, conventional techniques related to signalprocessing, data transmission, signaling, network control, and otherfunctional aspects of the systems (and the individual operatingcomponents of the systems) may not be described in detail herein.Furthermore, the connecting lines shown in the various figures containedherein are intended to represent exemplary functional relationshipsand/or physical couplings between the various elements. It should benoted that many alternative or additional functional relationships orphysical connections may be present in an embodiment of the subjectmatter.

The various tasks performed in connection with viewing, objectidentification, sharing and information retrieving processes between themobile client and agent in video-chat applications may be performed bysoftware, hardware, firmware, or any combination thereof. Forillustrative purposes, the following description of object capture,shared display, and process may refer to elements mentioned above inconnection with FIGS. 1-7. In practice, portions of process of FIGS. 1-7may be performed by different elements of the described system, e.g.,mobile clients, agents, in-app applications etc. It should beappreciated that process of FIGS. 1-7 may include any number ofadditional or alternative tasks, the tasks shown in FIGS. 1-7 need notbe performed in the illustrated order, and process of the FIGS. 1-7 maybe incorporated into a more comprehensive procedure or process havingadditional functionality not described in detail herein. Moreover, oneor more of the tasks shown in FIG. 1-7 could be omitted from anembodiment of the process shown in FIGS. 1-7 as long as the intendedoverall functionality remains intact.

The foregoing detailed description is merely illustrative in nature andis not intended to limit the embodiments of the subject matter or theapplication and uses of such embodiments. As used herein, the word“exemplary” means “serving as an example, instance, or illustration.”Any implementation described herein as exemplary is not necessarily tobe construed as preferred or advantageous over other implementations.Furthermore, there is no intention to be bound by any expressed orimplied theory presented in the preceding technical field, background,or detailed description.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or embodiments described herein are not intended tolimit the scope, applicability, or configuration of the claimed subjectmatter in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the described embodiment or embodiments. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the scope defined by theclaims, which includes known equivalents and foreseeable equivalents atthe time of filing this patent application.

1. A method for identifying object information in a video chatapplication hosted by a server system, comprising: presenting a sharedview for an agent device and a client device during a video chat sessionusing the video chat application hosted by the server system; receiving,by the server system, object images captured by the client device duringthe video chat session; receiving a touch selection of an object fromthe captured object images through the agent device during the videochat session; identifying, by the server system, the selected object bydetermining an approximate center of the touch selection; overlaying theselected object with an image color to identify the selected objectwithin the shared view of the video chat session; generating, by theserver system, object information to identify features of the selectedobject; sending generated object information of the selected object to adatabase system associated with the server system to search for andretrieve data related to the generated object information stored by thedatabase system; receiving, from the database system, data related tothe object information; and sending the received data related to theobject information from the server system to the client device.
 21. Themethod of claim 1, wherein the database system comprises a knowledgedatabase of data related to the object information.
 22. The method ofclaim 1, further comprising: capturing the images of the objects with acamera coupled to the client device associated with the server system.23. The method of claim 1, further comprising: identifying, by locationswithin the images by the server, the features of the objects in thesession of the video chat application.
 24. The method of claim 1,further comprising: receiving instructions by the server system toperform the search of the database system for retrieving the relateddata.
 25. The method of claim 24, further comprising: applyingartificial intelligence solutions, by the server system, to perform thesearch of the database system for retrieving the related data.
 26. Themethod of claim 1, further comprising: overlaying, by the server system,the features of the objects captured in a display of the server systemto distinguish the features from a background of the display.
 27. Themethod of claim 1, further comprising: identifying the features of theobject within an agent app hosted by the server system.
 28. The methodof claim 27, further comprising: identifying, by locations within theimages in the agent app by the server system, the features of theobjects in the session of the video chat application.
 29. A computerprogram product tangibly embodied in a computer-readable storage deviceand comprising instructions configurable to be executed by a processorto perform a method for processing requests by a server system in avideo chat application, the method comprising: presenting a shared viewfor an agent device and a client device during a video chat sessionusing the video chat application hosted by the server system; receiving,by the server system, object images captured by the client device duringthe video chat session; receiving a touch selection of an object fromthe captured object images through the agent device during the videochat session; identifying, by the server system, the selected object bydetermining an approximate center of the touch selection; overlaying theselected object with an image color to identify the selected objectwithin the shared view of the video chat session; generating, by theserver system, object information to identify features of the selectedobject; sending generated object information of the selected object to adatabase system associated with the server system to search for andretrieve data related to the generated object information stored by thedatabase system; receiving, from the database system, data related tothe object information; and sending the received data related to theobject information from the server system to the client device.
 30. Themethod of claim 29, wherein the database system comprises a knowledgedatabase of data related to the object information.
 31. The method ofclaim 29, further comprising: capturing the images of the objects with acamera coupled to the client device associated with the server system.32. The method of claim 29, further comprising: identifying, bylocations within the images by the server, the features of the objectsin the session of the video chat application.
 33. The method of claim29, further comprising: receiving instructions by the server system toperform the search of the database system for retrieving the relateddata.
 34. The method of claim 33, further comprising: applyingartificial intelligence solutions, by the server system, to perform thesearch of the database system for retrieving the related data.
 35. Themethod of claim 29, further comprising: overlaying, by the serversystem, the features of the objects captured in a display of the serversystem to distinguish the features from a background of the display. 36.The method of claim 29, the server system further comprising:identifying the features of the object within an agent app hosted by theserver system.
 37. The method of claim 36, further comprising:identifying, by locations within the images in the agent app by theserver system, the features of the objects in the session of the videochat application.
 38. A server system comprising: at least oneprocessor; a database system; and at least one computer-readable storagedevice comprising instructions configurable to be executed by the atleast one processor to perform a method for processing requests to thedatabase system using a video chat application, the method comprising:presenting a shared view for an agent device and a client device duringa video chat session using the video chat application hosted by theserver system; receiving, by the server system, object images capturedby the client device during the video chat session; receiving a touchselection of an object from the captured object images through the agentdevice during the video chat session; identifying, by the server system,the selected object by determining an approximate center of the touchselection; overlaying the selected object with an image color toidentify the selected object within the shared view of the video chatsession; generating, by the server system, object information toidentify features of the selected object; sending generated objectinformation of the selected object to a database system associated withthe server system to search for and retrieve data related to thegenerated object information stored by the database system; receiving,from the database system, data related to the object information; andsending the received data related to the object information from theserver system to the client device.
 39. The system of claim 38 whereinthe database system comprises a multi-tenant database system.